Loading data
library(statisticalModeling)
library(mosaicModel) # newer version of statisticalModeling with better functions
library(tidyverse)
library(mosaic)
library(ggplot2)
library(dplyr)
library(broom)
library(lattice)
library(gridExtra)
library(grid)
ads_sales <- as_tibble(read.csv("Advertising.csv"))
ads_sales
Oversimplified Modeling process
- Fit a model
- Evaluate the model by adding residuals and looking at MSE
- Use model for decision
Less naive modeling
- Exploratory data analysis
*Split data into training and test
- Fit a model
- Evaluate the model by adding residuals and looking at MSE *Using cross-validation
Interpret model *Using bootstrapped confidence intervals for effect sizes / parameters
- Use model for decision
Exploratory Data Analysis
Before doing any modeling, you should get a basic feel for dependent and indpendent variables and their relationships visually.
#look at the data with histograms and scatterplots
# Package
# this is a pedagogical package from a 2011 textbook
# with some useful graphics functions
# library(car)
# I'm not library'ing it b/c it has some namespace clashes with dplyr
car::scatterplotMatrix(ads_sales)

#let's make a nicer looking scatter plot matrix
# We should all ask ourselves why the makers of graphics libraries
# don't have better defaults...
# Package
# This is the R package version of the
# color brewer website we discussed for making accessible color schemes
library(RColorBrewer)
# Make the plot
car::scatterplotMatrix(~ sales + TV + radio + newspaper , data=ads_sales ,
#reg.line="" , #would turn regression lines off
#smoother="", #would turn Loess regression lines off
# col=my_colors ,
smoother.args=list(col="grey") ,
#cex=1.5 ,
#pch=c(15,16,17) ,
main="Ad Sales Scatter Plot Matrix")

# Another example with categorical data from mtcars dataset
# If you needed a color scheme for a categorical variable
# this is how you would make one with color brewer
my_colors <- brewer.pal(nlevels(as.factor(mtcars$cyl)), "Set2")
# example of how to make SPLOM with categorical coloring
car::scatterplotMatrix(~mpg+disp+drat|cyl, data=mtcars,
reg.line="" ,
smoother="",
col=my_colors ,
smoother.args=list(col="grey") ,
cex=1.5 ,
pch=c(15,16,17) ,
main="Scatter plot with Three Cylinder Options")

If we were doing a deeper analysis, we’d write some questions below and then answer them, to explore the data. Here are some examples from HW5, on data about faculty salaries.
# What is the number of males/females in the dataset? What does this already tell you...?
# What is the mean salary by sex? Hint: you'll have to groupby sex (`sx`)
# Draw histograms for the distribution of salaries for males and females (separately)
# Hint: you can use ggplot and facet
# The x and y axes should be consistent between the graphs
# Draw histograms for the distribution of salaries by rank
# Create scatterplots to show how salary compares to years since degree / in current rank
Simple linear regression:
First, make a model and visualize it.
# Create a simple linear model that assesses the relationship tv radio, and newspapers and sales
model <- lm ( sales ~ TV + radio + newspaper, data=ads_sales)
model
Call:
lm(formula = sales ~ TV + radio + newspaper, data = ads_sales)
Coefficients:
(Intercept) TV radio newspaper
2.938889 0.045765 0.188530 -0.001037
#old statisticalModeling way
fmodel(model)

#new mosaicModel way, use this one in your code
# Compare the old and new, and notice the new one has better information
# it really is an upgrade!
# To reinforce what I said in lecture, the reason we're using mosaicModel
# is to have more time to focus on the essential problems in modeling;
# these packages implement minutiae / detailed knowledge like the
# formula for calculating an effect size.
mod_plot(model)

Statistical summary of model
For most models in r, they provide a summary. You can broom the summary to get most of the information out in rows instead of a strange string format. Looking at this the p-value of the model is very low - the probability of these coefficients assuming the data was generated randomly is very low! The R^2 is also really high! Naively, we might think this means we have a great model. Far from it. This is why we need to look at residuals to check for model bias / systematic lack of fit with data.
# Interpret the metric for accuracy above.
summary(model)
Call:
lm(formula = sales ~ TV + radio + newspaper, data = ads_sales)
Residuals:
Min 1Q Median 3Q Max
-8.8277 -0.8908 0.2418 1.1893 2.8292
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 2.938889 0.311908 9.422 <2e-16 ***
TV 0.045765 0.001395 32.809 <2e-16 ***
radio 0.188530 0.008611 21.893 <2e-16 ***
newspaper -0.001037 0.005871 -0.177 0.86
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 1.686 on 196 degrees of freedom
Multiple R-squared: 0.8972, Adjusted R-squared: 0.8956
F-statistic: 570.3 on 3 and 196 DF, p-value: < 2.2e-16
tidy(summary(model))
Visualize the distribution of the predictions and actual values
If the predicted values have a different distribution, we might have model bias (ie the model architecture or formula doesn’t match the patterns in the data).
# add residuals and fitted values (ie the model's prediction)
sales_with_residuals <- augment(model, ads_sales)
Deprecated: please use `purrr::possibly()` insteadDeprecated: please use `purrr::possibly()` insteadDeprecated: please use `purrr::possibly()` insteadDeprecated: please use `purrr::possibly()` insteadDeprecated: please use `purrr::possibly()` instead
sales_with_residuals
#never compare histograms with different scales and bin-widths
# like these ones
hist(sales_with_residuals$.fitted)

hist(sales_with_residuals$sales)

# here we force them to have the same axes
binwide<-.5
g <- ggplot(data=sales_with_residuals, aes(sales))
g1 <- g + geom_histogram(
binwidth = binwide,
col="black",
size=.1) + xlim(c(0, 30)) + ylim(c(0, 30)) + labs(title=paste("Actual Sales, binwidth=",binwide))
g <- ggplot(data=sales_with_residuals, aes(.fitted))
g2 <- g + geom_histogram(
binwidth = binwide,
col="black",
size=.1) + xlim(c(0, 30)) + ylim(c(0, 30)) + labs(title=paste("Predicted Sales, binwidth=",binwide))
grid.arrange(g1,g2,ncol=1)

# now, binwidth can make a big difference in comparing plots,
# so we'll make a for loop that tries different widths
for(binwide in seq(.5,1, length.out=10)){
g <- ggplot(data=sales_with_residuals, aes(sales))
g1 <- g + geom_histogram(
binwidth = binwide,
col="black",
size=.1) + xlim(c(0, 30)) + ylim(c(0, 30)) + labs(title=paste("Actual Sales, binwidth=",binwide))
g <- ggplot(data=sales_with_residuals, aes(.fitted))
g2 <- g + geom_histogram(
binwidth = binwide,
col="black",
size=.1) + xlim(c(0, 30)) + ylim(c(0, 30)) + labs(title=paste("Predicted Sales, binwidth=",binwide))
grid.arrange(g1,g2,ncol=1)
}










Visualize the residuals of the model to check for model bias
If the residuals don’t look like random noise, we very likely have model bias, and should try different formulas or model architectures. Random noise for most models is Gaussian noise (the Gaussian distribution is also called the “normal” distribution, but I think that name is misleading. It’s very important as a distribution and useful, but the word “normal” makes it sound like it’s what data “usually” follows, which really is untrue.)
g <- ggplot(data=sales_with_residuals, aes(.resid))
g1 <- g + geom_histogram(
binwidth = .5,
col="black",
size=.1) + labs(subtitle="No interaction model")
g1

# let's get the limits of the plot by absolute value of the residuals
# this will help us see the residuals aren't symmetric around zero
# like gaussian random error should be / is assumed by linear model
plot_limit <- max(abs(sales_with_residuals$.resid))+.5
g <- ggplot(data=sales_with_residuals, aes(.resid))
sales_residuals_plot <- g + geom_histogram(
binwidth = .5,
col="black",
size=.1) + xlim(c(-plot_limit, plot_limit)) + labs(subtitle="No interaction model")
sales_residuals_plot

We can see here they don’t look like random noise.
Here’s what random Gaussian noise might look like. You can generate Gaussian noise with rnorm(mean,sd). I’m getting sd as the standard error from the model, this uses some statistical estimation under the hood.
sd <- sigma(model)
sd
[1] 1.68551
# the do(n) syntax is a shortcut for
# doing the following parts n times, then combining the results into a vector
nrow(ads_sales)
[1] 200
simulated_residuals_for_gaussian_noise <- do(nrow(ads_sales)) * rnorm(1,0,sd)
simulated_residuals_for_gaussian_noise
g <- ggplot(data=simulated_residuals_for_gaussian_noise, aes(rnorm))
simulated_residuals_for_gaussian_noise_1_plot <- g + geom_histogram(
binwidth = .5,
col="black",
size=.1) + xlim(c(-plot_limit, plot_limit)) + labs(subtitle="Simulated Gaussian noise")
grid.arrange(simulated_residuals_for_gaussian_noise_1_plot,sales_residuals_plot,ncol=1)

Look at summary statistics for the model’s accuracy, like MSE
Here we aren’t using cross-validation. In practice you should always calculate this using cross-validation. You can even do the steps above 10x with cross validation, showing residuals and predictions on the testing dataset; a very prudent and careful modeler might do that in practice.
#Evaluate the accuracy of your model. Calculate a metric for it.
# this is the statiscalModeling function
# they want you to use cross validation so badly (and you should)
# that they don't have a separate function for MSE without cross-validation
# cv_pred_error(model)
# Here's the "new" way with mosaicModel.
mod_error(model) #MSE
Calculating error from training data.
mse
2.784126
# you can look at broom(summary(model)) to get
# a more statistically "accurate" estimation of SE
# though those methods assume the model's formula and architecture match reality
sqrt(mod_error(model))
Calculating error from training data.
mse
1.66857
Next, we start interpreting the model
We’re doing this as a part of showing the new mosaicModel functions, which correspond to the statisticalModeling functions you learned online. Normally, if the residuals didn’t look good, then we wouldn’t interpret the model, b/c the data doesn’t appear to be fit well by the model (its assumptions appear violated).
# What is the effect size of each explanatory variable on the outcome variable (dependent variable)?
#old way
# effect_size(model, ~ variables to change)
effect_size(model, ~ TV)
#new way
mod_effect(model, ~ TV)
mod_effect(model, ~ radio)
mod_effect(model, ~ newspaper)
# here's the underlying way mod_effect works
effect_size_of_newspaper <- (sales_modeled(TV=150, radio=23, newspaper = 46)$model_output -sales_modeled(TV=150, radio=23, newspaper = 26)$model_output) / (46-26)
effect_size_of_newspaper
[1] -0.001037493
# here's how to get the function for the model
# i.e. function(explanatory variables) which gives a prediction for dependent variable
sales_modeled <- mod_fun(model)
Skeptical Modeling process
Here’s a process to follow when modeling (as a refresher). We won’t cover all the steps here in this analysis.
- Exploratory data analysis
Split data into training and test
- Fit a model
- Evaluate the model by adding residuals and looking at MSE Using cross-validation
Interpret model Using bootstrapped confidence intervals for effect sizes / parameters
- Use model for decision
Continuing our analysis
As the model above didn’t seem to fit the data, let’s try one with an interaction term, motivated by our domain knowledge.
# Using multiple regression with interaction terms:
model_2 <- lm ( sales ~ TV * radio + newspaper, data=ads_sales)
model_2
Call:
lm(formula = sales ~ TV * radio + newspaper, data = ads_sales)
Coefficients:
(Intercept) TV radio newspaper TV:radio
6.728412 0.019067 0.027992 0.001444 0.001087
# let's compare the two models, the new one has interaction effects
grid.arrange(mod_plot(model)+labs(subtitle="No interactions model") ,
mod_plot(model_2)+labs(subtitle="TV*radio + newspaper model (with interaction)"), ncol=1)

# What is the effect size of each explanatory variable on the outcome variable (dependent variable)?
mod_effect(model, ~ TV) # like effect_size
mod_effect(model_2, ~ TV) # like effect_size
mod_effect(model_2, ~ radio)
mod_effect(model_2, ~ newspaper)
sales_model_2ed <- mod_fun(model_2)
# sales = a1 + a2*tv + a3*radio + a4*tv*radio
effect_size_of_newspaper <- (sales_model_2ed(TV=150, radio=23, newspaper = 46)$model_2_output -sales_model_2ed(TV=150, radio=23, newspaper = 26)$model_2_output) / (46-26)
effect_size_of_newspaper
numeric(0)
# Hint: you can use tools from statisticalModeling package used in the Datacamp course
# Evaluate the accuracy of your model_2. Calculate a metric for it.
sqrt(mod_error(model_2))
Calculating error from training data.
mse
0.933573
hist(ads_sales$sales)

# Interpret the metric for accuracy above.
summary(model_2)
Call:
lm(formula = sales ~ TV * radio + newspaper, data = ads_sales)
Residuals:
Min 1Q Median 3Q Max
-6.2929 -0.3983 0.1811 0.5957 1.5009
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 6.728e+00 2.533e-01 26.561 < 2e-16 ***
TV 1.907e-02 1.509e-03 12.633 < 2e-16 ***
radio 2.799e-02 9.141e-03 3.062 0.00251 **
newspaper 1.444e-03 3.295e-03 0.438 0.66169
TV:radio 1.087e-03 5.256e-05 20.686 < 2e-16 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 0.9455 on 195 degrees of freedom
Multiple R-squared: 0.9678, Adjusted R-squared: 0.9672
F-statistic: 1466 on 4 and 195 DF, p-value: < 2.2e-16
sales_with_residuals_2 <- augment(model_2, ads_sales)
Deprecated: please use `purrr::possibly()` insteadDeprecated: please use `purrr::possibly()` insteadDeprecated: please use `purrr::possibly()` insteadDeprecated: please use `purrr::possibly()` insteadDeprecated: please use `purrr::possibly()` instead
sales_with_residuals_2
g <- ggplot(data=sales_with_residuals_2, aes(sales))
g + geom_histogram(
binwidth = .5,
col="black",
size=.1) + xlim(c(0, 30))

g <- ggplot(data=sales_with_residuals_2, aes(.fitted))
g + geom_histogram(
binwidth = .5,
col="black",
size=.1) + xlim(c(0, 30))

g <- ggplot(data=sales_with_residuals, aes(.fitted))
g + geom_histogram(
binwidth = .5,
col="black",
size=.1) + xlim(c(0, 30)) + labs(subtitle="No interaction model")

sales_with_residuals_2 <- augment(model_2, ads_sales)
Deprecated: please use `purrr::possibly()` insteadDeprecated: please use `purrr::possibly()` insteadDeprecated: please use `purrr::possibly()` insteadDeprecated: please use `purrr::possibly()` insteadDeprecated: please use `purrr::possibly()` instead
sales_with_residuals_2
# residuals histograms
plot_limit <- max(abs(sales_with_residuals_2$.resid))+.5
g <- ggplot(data=sales_with_residuals_2, aes(.resid))
sales_residuals_plot_2 <- g + geom_histogram(
binwidth = .5,
col="black",
size=.1) + xlim(c(-plot_limit, plot_limit)) + labs(subtitle=" Has interaction model")
sales_residuals_plot_2

# compare to simulated residuals
sd2 <- sigma(model_2)
sd2
[1] 0.9454661
# the do(n) syntax is a shortcut for
# doing the following parts n times, then combining the results into a vector
nrow(ads_sales)
[1] 200
simulated_residuals_for_gaussian_noise2 <- do(nrow(ads_sales)) * rnorm(1,0,sd2)
simulated_residuals_for_gaussian_noise2
g <- ggplot(data=simulated_residuals_for_gaussian_noise2, aes(rnorm))
g <- g + geom_histogram(
binwidth = .5,
col="black",
size=.1) + xlim(c(-plot_limit, plot_limit)) + labs(subtitle="Simulated Gaussian noise")
grid.arrange(g,sales_residuals_plot_2,ncol=1)

#compare to the no interaction model
grid.arrange(simulated_residuals_for_gaussian_noise_1_plot,sales_residuals_plot,ncol=1)

# you can see that our new model has better looking residuals, more similar to
# gaussian noise
# modeling proceeds in a sequence like this
# 1. try model, evaluate its fit
# 2. change model architecture and formula when you see patterns in residuals
# 3. repeat, to the quality of model you need for your purpose (strive for higher tho)
Assess predictions
Here are some built-in plots that all have the same purpose - to validate that the model fits your data. I won’t go in depth with these, but the general idea of each is that you want the points to fall around the dotted line with just random variation around it without any patterns (on q-q plot , the 2nd one from the plot call, you want them to fall on the line).
There are also statistical tests for normality of residuals, but in practice almost no one uses them because 1) they almost always say that the residuals are not normal (which is a blow to your model so they just ignore that…) 2) I have heard people say that the linear model has some “robustness” against deviations from its assumptions, but I don’t really buy that b/c if we understood the robustness conditions well, we should be able to make a test for “normal enough” residuals, and I’ve never seen one or heard one mentioned. Thus, I’m emphasizing the visual approach (which is also more informative than a p-value).
plot(model_2)




#compare to model without interactions
plot(model)




LS0tCnRpdGxlOiAiQ2xhc3MgNi4xIEV4YW1wbGUgaW4gY2xhc3MgLSBMaW5lYXIgbW9kZWxpbmcgd2l0aCBjb21tZW50YXJ5IgpvdXRwdXQ6CiAgaHRtbF9kb2N1bWVudDogZGVmYXVsdAogIGh0bWxfbm90ZWJvb2s6IGRlZmF1bHQKLS0tCgpMb2FkaW5nIGRhdGEKYGBge3J9CmxpYnJhcnkoc3RhdGlzdGljYWxNb2RlbGluZykKbGlicmFyeShtb3NhaWNNb2RlbCkgIyBuZXdlciB2ZXJzaW9uIG9mIHN0YXRpc3RpY2FsTW9kZWxpbmcgd2l0aCBiZXR0ZXIgZnVuY3Rpb25zCmxpYnJhcnkodGlkeXZlcnNlKQpsaWJyYXJ5KG1vc2FpYykKbGlicmFyeShnZ3Bsb3QyKQpsaWJyYXJ5KGRwbHlyKQpsaWJyYXJ5KGJyb29tKQpsaWJyYXJ5KGxhdHRpY2UpCmxpYnJhcnkoZ3JpZEV4dHJhKQpsaWJyYXJ5KGdyaWQpCgphZHNfc2FsZXMgPC0gYXNfdGliYmxlKHJlYWQuY3N2KCJBZHZlcnRpc2luZy5jc3YiKSkKYWRzX3NhbGVzCmBgYAoKIyMgT3ZlcnNpbXBsaWZpZWQgTW9kZWxpbmcgcHJvY2VzcwoKMS4gRml0IGEgbW9kZWwKMi4gRXZhbHVhdGUgdGhlIG1vZGVsIGJ5IGFkZGluZyByZXNpZHVhbHMgYW5kIGxvb2tpbmcgYXQgTVNFCjMuIFVzZSBtb2RlbCBmb3IgZGVjaXNpb24KCgojIExlc3MgbmFpdmUgbW9kZWxpbmcKMS4gRXhwbG9yYXRvcnkgZGF0YSBhbmFseXNpcwoKKlNwbGl0IGRhdGEgaW50byB0cmFpbmluZyBhbmQgdGVzdAoKMi4gRml0IGEgbW9kZWwKMy4gRXZhbHVhdGUgdGhlIG1vZGVsIGJ5IGFkZGluZyByZXNpZHVhbHMgYW5kIGxvb2tpbmcgYXQgTVNFCiAgKlVzaW5nIGNyb3NzLXZhbGlkYXRpb24KICAKSW50ZXJwcmV0IG1vZGVsCiAgKlVzaW5nIGJvb3RzdHJhcHBlZCBjb25maWRlbmNlIGludGVydmFscyBmb3IgZWZmZWN0IHNpemVzIC8gcGFyYW1ldGVycwogIAo0LiBVc2UgbW9kZWwgZm9yIGRlY2lzaW9uCgojIyBFeHBsb3JhdG9yeSBEYXRhIEFuYWx5c2lzCkJlZm9yZSBkb2luZyBhbnkgbW9kZWxpbmcsIHlvdSBzaG91bGQgZ2V0IGEgYmFzaWMgZmVlbCBmb3IgZGVwZW5kZW50IGFuZCBpbmRwZW5kZW50IHZhcmlhYmxlcyBhbmQgdGhlaXIgcmVsYXRpb25zaGlwcyB2aXN1YWxseS4KYGBge3J9CiNsb29rIGF0IHRoZSBkYXRhIHdpdGggaGlzdG9ncmFtcyBhbmQgc2NhdHRlcnBsb3RzCgojIFBhY2thZ2UKIyB0aGlzIGlzIGEgcGVkYWdvZ2ljYWwgcGFja2FnZSBmcm9tIGEgMjAxMSB0ZXh0Ym9vawojIHdpdGggc29tZSB1c2VmdWwgZ3JhcGhpY3MgZnVuY3Rpb25zCiMgbGlicmFyeShjYXIpCiMgSSdtIG5vdCBsaWJyYXJ5J2luZyBpdCBiL2MgaXQgaGFzIHNvbWUgbmFtZXNwYWNlIGNsYXNoZXMgd2l0aCBkcGx5cgoKCmNhcjo6c2NhdHRlcnBsb3RNYXRyaXgoYWRzX3NhbGVzKQoKI2xldCdzIG1ha2UgYSBuaWNlciBsb29raW5nIHNjYXR0ZXIgcGxvdCBtYXRyaXgKIyAgV2Ugc2hvdWxkIGFsbCBhc2sgb3Vyc2VsdmVzIHdoeSB0aGUgbWFrZXJzIG9mIGdyYXBoaWNzIGxpYnJhcmllcwojICBkb24ndCBoYXZlIGJldHRlciBkZWZhdWx0cy4uLgoKIyBQYWNrYWdlCiMgVGhpcyBpcyB0aGUgUiBwYWNrYWdlIHZlcnNpb24gb2YgdGhlCiMgY29sb3IgYnJld2VyIHdlYnNpdGUgd2UgZGlzY3Vzc2VkIGZvciBtYWtpbmcgYWNjZXNzaWJsZSBjb2xvciBzY2hlbWVzCmxpYnJhcnkoUkNvbG9yQnJld2VyKQoKIyBNYWtlIHRoZSBwbG90CmNhcjo6c2NhdHRlcnBsb3RNYXRyaXgofiBzYWxlcyArIFRWICsgcmFkaW8gKyBuZXdzcGFwZXIgLCBkYXRhPWFkc19zYWxlcyAsIAogICAgICAgICAgICAgICAgICAjcmVnLmxpbmU9IiIgLCAjd291bGQgdHVybiByZWdyZXNzaW9uIGxpbmVzIG9mZgogICAgICAgICAgICAgICAgICAjc21vb3RoZXI9IiIsICN3b3VsZCB0dXJuIExvZXNzIHJlZ3Jlc3Npb24gbGluZXMgb2ZmCiAgICAgICAgICAgICAgICAgICMgY29sPW15X2NvbG9ycyAsCiAgICAgICAgICAgICAgICAgIHNtb290aGVyLmFyZ3M9bGlzdChjb2w9ImdyZXkiKSAsIAogICAgICAgICAgICAgICAgICAjY2V4PTEuNSAsIAogICAgICAgICAgICAgICAgICAjcGNoPWMoMTUsMTYsMTcpICwgCiAgICAgICAgICAgICAgICAgIG1haW49IkFkIFNhbGVzIFNjYXR0ZXIgUGxvdCBNYXRyaXgiKQoKIyBBbm90aGVyIGV4YW1wbGUgd2l0aCBjYXRlZ29yaWNhbCBkYXRhIGZyb20gbXRjYXJzIGRhdGFzZXQKCiMgSWYgeW91IG5lZWRlZCBhIGNvbG9yIHNjaGVtZSBmb3IgYSBjYXRlZ29yaWNhbCB2YXJpYWJsZQojIHRoaXMgaXMgaG93IHlvdSB3b3VsZCBtYWtlIG9uZSB3aXRoIGNvbG9yIGJyZXdlcgpteV9jb2xvcnMgPC0gYnJld2VyLnBhbChubGV2ZWxzKGFzLmZhY3RvcihtdGNhcnMkY3lsKSksICJTZXQyIikKCiMgZXhhbXBsZSBvZiBob3cgdG8gbWFrZSBTUExPTSB3aXRoIGNhdGVnb3JpY2FsIGNvbG9yaW5nCmNhcjo6c2NhdHRlcnBsb3RNYXRyaXgofm1wZytkaXNwK2RyYXR8Y3lsLCBkYXRhPW10Y2FycywgCiAgICAgICAgICAgICAgICAgIHJlZy5saW5lPSIiICwgCiAgICAgICAgICAgICAgICAgIHNtb290aGVyPSIiLCAKICAgICAgICAgICAgICAgICAgIGNvbD1teV9jb2xvcnMgLAogICAgICAgICAgICAgICAgICBzbW9vdGhlci5hcmdzPWxpc3QoY29sPSJncmV5IikgLCAKICAgICAgICAgICAgICAgICAgY2V4PTEuNSAsIAogICAgICAgICAgICAgICAgICBwY2g9YygxNSwxNiwxNykgLCAKICAgICAgICAgICAgICAgICAgbWFpbj0iU2NhdHRlciBwbG90IHdpdGggVGhyZWUgQ3lsaW5kZXIgT3B0aW9ucyIpCgoKYGBgCklmIHdlIHdlcmUgZG9pbmcgYSBkZWVwZXIgYW5hbHlzaXMsIHdlJ2Qgd3JpdGUgc29tZSBxdWVzdGlvbnMgYmVsb3cgYW5kIHRoZW4gYW5zd2VyIHRoZW0sIHRvIGV4cGxvcmUgdGhlIGRhdGEuIEhlcmUgYXJlIHNvbWUgZXhhbXBsZXMgZnJvbSBIVzUsIG9uIGRhdGEgYWJvdXQgZmFjdWx0eSBzYWxhcmllcy4KYGBge3J9CgojIFdoYXQgaXMgdGhlIG51bWJlciBvZiBtYWxlcy9mZW1hbGVzIGluIHRoZSBkYXRhc2V0PyBXaGF0IGRvZXMgdGhpcyBhbHJlYWR5IHRlbGwgeW91Li4uPwoKIyBXaGF0IGlzIHRoZSBtZWFuIHNhbGFyeSBieSBzZXg/IEhpbnQ6IHlvdSdsbCBoYXZlIHRvIGdyb3VwYnkgc2V4IChgc3hgKQoKIyBEcmF3IGhpc3RvZ3JhbXMgZm9yIHRoZSBkaXN0cmlidXRpb24gb2Ygc2FsYXJpZXMgZm9yIG1hbGVzIGFuZCBmZW1hbGVzIChzZXBhcmF0ZWx5KQojIEhpbnQ6IHlvdSBjYW4gdXNlIGdncGxvdCBhbmQgZmFjZXQgCiMgVGhlIHggYW5kIHkgYXhlcyBzaG91bGQgYmUgY29uc2lzdGVudCBiZXR3ZWVuIHRoZSBncmFwaHMKCiMgRHJhdyBoaXN0b2dyYW1zIGZvciB0aGUgZGlzdHJpYnV0aW9uIG9mIHNhbGFyaWVzIGJ5IHJhbmsKCiMgQ3JlYXRlIHNjYXR0ZXJwbG90cyB0byBzaG93IGhvdyBzYWxhcnkgY29tcGFyZXMgdG8geWVhcnMgc2luY2UgZGVncmVlIC8gaW4gY3VycmVudCByYW5rCgpgYGAKIyNTaW1wbGUgbGluZWFyIHJlZ3Jlc3Npb246CkZpcnN0LCBtYWtlIGEgbW9kZWwgYW5kIHZpc3VhbGl6ZSBpdC4KYGBge3J9CiMgQ3JlYXRlIGEgc2ltcGxlIGxpbmVhciBtb2RlbCB0aGF0IGFzc2Vzc2VzIHRoZSByZWxhdGlvbnNoaXAgdHYgcmFkaW8sIGFuZCBuZXdzcGFwZXJzIGFuZCBzYWxlcwptb2RlbCA8LSBsbSAoIHNhbGVzIH4gVFYgKyByYWRpbyArIG5ld3NwYXBlciwgZGF0YT1hZHNfc2FsZXMpCm1vZGVsCgojb2xkIHN0YXRpc3RpY2FsTW9kZWxpbmcgd2F5CmZtb2RlbChtb2RlbCkKI25ldyBtb3NhaWNNb2RlbCB3YXksIHVzZSB0aGlzIG9uZSBpbiB5b3VyIGNvZGUKIyBDb21wYXJlIHRoZSBvbGQgYW5kIG5ldywgYW5kIG5vdGljZSB0aGUgbmV3IG9uZSBoYXMgYmV0dGVyIGluZm9ybWF0aW9uCiMgaXQgcmVhbGx5IGlzIGFuIHVwZ3JhZGUhCiMgICBUbyByZWluZm9yY2Ugd2hhdCBJIHNhaWQgaW4gbGVjdHVyZSwgdGhlIHJlYXNvbiB3ZSdyZSB1c2luZyBtb3NhaWNNb2RlbCAKIyAgIGlzIHRvIGhhdmUgbW9yZSB0aW1lIHRvIGZvY3VzIG9uIHRoZSBlc3NlbnRpYWwgcHJvYmxlbXMgaW4gbW9kZWxpbmc7CiMgICB0aGVzZSBwYWNrYWdlcyBpbXBsZW1lbnQgbWludXRpYWUgLyBkZXRhaWxlZCBrbm93bGVkZ2UgbGlrZSB0aGUgCiMgICBmb3JtdWxhIGZvciBjYWxjdWxhdGluZyBhbiBlZmZlY3Qgc2l6ZS4KbW9kX3Bsb3QobW9kZWwpIApgYGAKIyBTdGF0aXN0aWNhbCBzdW1tYXJ5IG9mIG1vZGVsCkZvciBtb3N0IG1vZGVscyBpbiByLCB0aGV5IHByb3ZpZGUgYSBzdW1tYXJ5LiBZb3UgY2FuIGJyb29tIHRoZSBzdW1tYXJ5IHRvIGdldCBtb3N0IG9mIHRoZSBpbmZvcm1hdGlvbiBvdXQgaW4gcm93cyBpbnN0ZWFkIG9mIGEgc3RyYW5nZSBzdHJpbmcgZm9ybWF0LiBMb29raW5nIGF0IHRoaXMgdGhlIHAtdmFsdWUgb2YgdGhlIG1vZGVsIGlzIHZlcnkgbG93IC0gdGhlIHByb2JhYmlsaXR5IG9mIHRoZXNlIGNvZWZmaWNpZW50cyBhc3N1bWluZyB0aGUgZGF0YSB3YXMgZ2VuZXJhdGVkIHJhbmRvbWx5IGlzIHZlcnkgbG93ISBUaGUgUl4yIGlzIGFsc28gcmVhbGx5IGhpZ2ghIE5haXZlbHksIHdlIG1pZ2h0IHRoaW5rIHRoaXMgbWVhbnMgd2UgaGF2ZSBhIGdyZWF0IG1vZGVsLiBGYXIgZnJvbSBpdC4gVGhpcyBpcyB3aHkgd2UgbmVlZCB0byBsb29rIGF0IHJlc2lkdWFscyB0byBjaGVjayBmb3IgbW9kZWwgYmlhcyAvIHN5c3RlbWF0aWMgbGFjayBvZiBmaXQgd2l0aCBkYXRhLgpgYGB7cn0KIyBJbnRlcnByZXQgdGhlIG1ldHJpYyBmb3IgYWNjdXJhY3kgYWJvdmUuCnN1bW1hcnkobW9kZWwpCgp0aWR5KHN1bW1hcnkobW9kZWwpKQpgYGAKCiMgVmlzdWFsaXplIHRoZSBkaXN0cmlidXRpb24gb2YgdGhlIHByZWRpY3Rpb25zIGFuZCBhY3R1YWwgdmFsdWVzCklmIHRoZSBwcmVkaWN0ZWQgdmFsdWVzIGhhdmUgYSBkaWZmZXJlbnQgZGlzdHJpYnV0aW9uLCB3ZSBtaWdodCBoYXZlIG1vZGVsIGJpYXMgKGllIHRoZSBtb2RlbCBhcmNoaXRlY3R1cmUgb3IgZm9ybXVsYSBkb2Vzbid0IG1hdGNoIHRoZSBwYXR0ZXJucyBpbiB0aGUgZGF0YSkuCgpgYGB7cn0KIyBhZGQgcmVzaWR1YWxzIGFuZCBmaXR0ZWQgdmFsdWVzIChpZSB0aGUgbW9kZWwncyBwcmVkaWN0aW9uKQpzYWxlc193aXRoX3Jlc2lkdWFscyA8LSBhdWdtZW50KG1vZGVsLCBhZHNfc2FsZXMpCnNhbGVzX3dpdGhfcmVzaWR1YWxzCgojbmV2ZXIgY29tcGFyZSBoaXN0b2dyYW1zIHdpdGggZGlmZmVyZW50IHNjYWxlcyBhbmQgYmluLXdpZHRocwojIGxpa2UgdGhlc2Ugb25lcwpoaXN0KHNhbGVzX3dpdGhfcmVzaWR1YWxzJC5maXR0ZWQpCmhpc3Qoc2FsZXNfd2l0aF9yZXNpZHVhbHMkc2FsZXMpCgojIGhlcmUgd2UgZm9yY2UgdGhlbSB0byBoYXZlIHRoZSBzYW1lIGF4ZXMKYmlud2lkZTwtLjUKIGcgPC0gZ2dwbG90KGRhdGE9c2FsZXNfd2l0aF9yZXNpZHVhbHMsIGFlcyhzYWxlcykpIAogIAoKICBnMSA8LSBnICsgZ2VvbV9oaXN0b2dyYW0oIAogICAgYmlud2lkdGggPSBiaW53aWRlLCAKICAgIGNvbD0iYmxhY2siLCAKICAgIHNpemU9LjEpICsgeGxpbShjKDAsIDMwKSkgKyB5bGltKGMoMCwgMzApKSArIGxhYnModGl0bGU9cGFzdGUoIkFjdHVhbCBTYWxlcywgYmlud2lkdGg9IixiaW53aWRlKSkKICAKICBnIDwtIGdncGxvdChkYXRhPXNhbGVzX3dpdGhfcmVzaWR1YWxzLCBhZXMoLmZpdHRlZCkpIAogIGcyIDwtIGcgKyBnZW9tX2hpc3RvZ3JhbSggCiAgICBiaW53aWR0aCA9IGJpbndpZGUsIAogICAgY29sPSJibGFjayIsIAogICAgc2l6ZT0uMSkgKyB4bGltKGMoMCwgMzApKSArIHlsaW0oYygwLCAzMCkpICsgbGFicyh0aXRsZT1wYXN0ZSgiUHJlZGljdGVkIFNhbGVzLCBiaW53aWR0aD0iLGJpbndpZGUpKQogIAogIGdyaWQuYXJyYW5nZShnMSxnMixuY29sPTEpCiAgCiMgbm93LCBiaW53aWR0aCBjYW4gbWFrZSBhIGJpZyBkaWZmZXJlbmNlIGluIGNvbXBhcmluZyBwbG90cywKIyBzbyB3ZSdsbCBtYWtlIGEgZm9yIGxvb3AgdGhhdCB0cmllcyBkaWZmZXJlbnQgd2lkdGhzCmZvcihiaW53aWRlIGluIHNlcSguNSwxLCBsZW5ndGgub3V0PTEwKSl7CiAgCiAgZyA8LSBnZ3Bsb3QoZGF0YT1zYWxlc193aXRoX3Jlc2lkdWFscywgYWVzKHNhbGVzKSkgCiAgCgogIGcxIDwtIGcgKyBnZW9tX2hpc3RvZ3JhbSggCiAgICBiaW53aWR0aCA9IGJpbndpZGUsIAogICAgY29sPSJibGFjayIsIAogICAgc2l6ZT0uMSkgKyB4bGltKGMoMCwgMzApKSArIHlsaW0oYygwLCAzMCkpICsgbGFicyh0aXRsZT1wYXN0ZSgiQWN0dWFsIFNhbGVzLCBiaW53aWR0aD0iLGJpbndpZGUpKQogIAogIGcgPC0gZ2dwbG90KGRhdGE9c2FsZXNfd2l0aF9yZXNpZHVhbHMsIGFlcyguZml0dGVkKSkgCiAgZzIgPC0gZyArIGdlb21faGlzdG9ncmFtKCAKICAgIGJpbndpZHRoID0gYmlud2lkZSwgCiAgICBjb2w9ImJsYWNrIiwgCiAgICBzaXplPS4xKSArIHhsaW0oYygwLCAzMCkpICsgeWxpbShjKDAsIDMwKSkgKyBsYWJzKHRpdGxlPXBhc3RlKCJQcmVkaWN0ZWQgU2FsZXMsIGJpbndpZHRoPSIsYmlud2lkZSkpCiAgCiAgZ3JpZC5hcnJhbmdlKGcxLGcyLG5jb2w9MSkKICAKfQoKYGBgCgoKIyBWaXN1YWxpemUgdGhlIHJlc2lkdWFscyBvZiB0aGUgbW9kZWwgdG8gY2hlY2sgZm9yIG1vZGVsIGJpYXMKSWYgdGhlIHJlc2lkdWFscyBkb24ndCBsb29rIGxpa2UgcmFuZG9tIG5vaXNlLCB3ZSB2ZXJ5IGxpa2VseSBoYXZlIG1vZGVsIGJpYXMsIGFuZCBzaG91bGQgdHJ5IGRpZmZlcmVudCBmb3JtdWxhcyBvciBtb2RlbCBhcmNoaXRlY3R1cmVzLiBSYW5kb20gbm9pc2UgZm9yIG1vc3QgbW9kZWxzIGlzIEdhdXNzaWFuIG5vaXNlICh0aGUgR2F1c3NpYW4gZGlzdHJpYnV0aW9uIGlzIGFsc28gY2FsbGVkIHRoZSAibm9ybWFsIiBkaXN0cmlidXRpb24sIGJ1dCBJIHRoaW5rIHRoYXQgbmFtZSBpcyBtaXNsZWFkaW5nLiBJdCdzIHZlcnkgaW1wb3J0YW50IGFzIGEgZGlzdHJpYnV0aW9uIGFuZCB1c2VmdWwsIGJ1dCB0aGUgd29yZCAibm9ybWFsIiBtYWtlcyBpdCBzb3VuZCBsaWtlIGl0J3Mgd2hhdCBkYXRhICJ1c3VhbGx5IiBmb2xsb3dzLCB3aGljaCByZWFsbHkgaXMgdW50cnVlLikgIAoKYGBge3J9CgpnIDwtIGdncGxvdChkYXRhPXNhbGVzX3dpdGhfcmVzaWR1YWxzLCBhZXMoLnJlc2lkKSkgCmcxIDwtIGcgKyBnZW9tX2hpc3RvZ3JhbSggCiAgICAgICAgICAgICAgICAgICBiaW53aWR0aCA9IC41LCAKICAgICAgICAgICAgICAgICAgIGNvbD0iYmxhY2siLCAKICAgICAgICAgICAgICAgICAgIHNpemU9LjEpICsgbGFicyhzdWJ0aXRsZT0iTm8gaW50ZXJhY3Rpb24gbW9kZWwiKQpnMQoKIyBsZXQncyBnZXQgdGhlIGxpbWl0cyBvZiB0aGUgcGxvdCBieSBhYnNvbHV0ZSB2YWx1ZSBvZiB0aGUgcmVzaWR1YWxzCiMgdGhpcyB3aWxsIGhlbHAgdXMgc2VlIHRoZSByZXNpZHVhbHMgYXJlbid0IHN5bW1ldHJpYyBhcm91bmQgemVybwojIGxpa2UgZ2F1c3NpYW4gcmFuZG9tIGVycm9yIHNob3VsZCBiZSAvIGlzIGFzc3VtZWQgYnkgbGluZWFyIG1vZGVsCnBsb3RfbGltaXQgPC0gbWF4KGFicyhzYWxlc193aXRoX3Jlc2lkdWFscyQucmVzaWQpKSsuNQpnIDwtIGdncGxvdChkYXRhPXNhbGVzX3dpdGhfcmVzaWR1YWxzLCBhZXMoLnJlc2lkKSkgCnNhbGVzX3Jlc2lkdWFsc19wbG90IDwtIGcgKyBnZW9tX2hpc3RvZ3JhbSggCiAgICAgICAgICAgICAgICAgICBiaW53aWR0aCA9IC41LCAKICAgICAgICAgICAgICAgICAgIGNvbD0iYmxhY2siLCAKICAgICAgICAgICAgICAgICAgIHNpemU9LjEpICsgeGxpbShjKC1wbG90X2xpbWl0LCBwbG90X2xpbWl0KSkgKyBsYWJzKHN1YnRpdGxlPSJObyBpbnRlcmFjdGlvbiBtb2RlbCIpCnNhbGVzX3Jlc2lkdWFsc19wbG90CgpgYGAKV2UgY2FuIHNlZSBoZXJlIHRoZXkgZG9uJ3QgbG9vayBsaWtlIHJhbmRvbSBub2lzZS4gCgpIZXJlJ3Mgd2hhdCByYW5kb20gR2F1c3NpYW4gbm9pc2UgbWlnaHQgbG9vayBsaWtlLiBZb3UgY2FuIGdlbmVyYXRlIEdhdXNzaWFuIG5vaXNlIHdpdGggcm5vcm0obWVhbixzZCkuIEknbSBnZXR0aW5nIHNkIGFzIHRoZSBzdGFuZGFyZCBlcnJvciBmcm9tIHRoZSBtb2RlbCwgdGhpcyB1c2VzIHNvbWUgc3RhdGlzdGljYWwgZXN0aW1hdGlvbiB1bmRlciB0aGUgaG9vZC4KYGBge3J9CnNkIDwtIHNpZ21hKG1vZGVsKQpzZAojIHRoZSBkbyhuKSBzeW50YXggaXMgYSBzaG9ydGN1dCBmb3IKIyBkb2luZyB0aGUgZm9sbG93aW5nIHBhcnRzIG4gdGltZXMsIHRoZW4gY29tYmluaW5nIHRoZSByZXN1bHRzIGludG8gYSB2ZWN0b3IKbnJvdyhhZHNfc2FsZXMpCnNpbXVsYXRlZF9yZXNpZHVhbHNfZm9yX2dhdXNzaWFuX25vaXNlIDwtIGRvKG5yb3coYWRzX3NhbGVzKSkgKiBybm9ybSgxLDAsc2QpCnNpbXVsYXRlZF9yZXNpZHVhbHNfZm9yX2dhdXNzaWFuX25vaXNlCgpnIDwtIGdncGxvdChkYXRhPXNpbXVsYXRlZF9yZXNpZHVhbHNfZm9yX2dhdXNzaWFuX25vaXNlLCBhZXMocm5vcm0pKSAKc2ltdWxhdGVkX3Jlc2lkdWFsc19mb3JfZ2F1c3NpYW5fbm9pc2VfMV9wbG90IDwtIGcgKyBnZW9tX2hpc3RvZ3JhbSggCiAgICAgICAgICAgICAgICAgICBiaW53aWR0aCA9IC41LCAKICAgICAgICAgICAgICAgICAgIGNvbD0iYmxhY2siLCAKICAgICAgICAgICAgICAgICAgIHNpemU9LjEpICsgeGxpbShjKC1wbG90X2xpbWl0LCBwbG90X2xpbWl0KSkgKyBsYWJzKHN1YnRpdGxlPSJTaW11bGF0ZWQgR2F1c3NpYW4gbm9pc2UiKQoKIGdyaWQuYXJyYW5nZShzaW11bGF0ZWRfcmVzaWR1YWxzX2Zvcl9nYXVzc2lhbl9ub2lzZV8xX3Bsb3Qsc2FsZXNfcmVzaWR1YWxzX3Bsb3QsbmNvbD0xKQoKYGBgCgoKIyBMb29rIGF0IHN1bW1hcnkgc3RhdGlzdGljcyBmb3IgdGhlIG1vZGVsJ3MgYWNjdXJhY3ksIGxpa2UgTVNFCkhlcmUgd2UgYXJlbid0IHVzaW5nIGNyb3NzLXZhbGlkYXRpb24uIEluIHByYWN0aWNlIHlvdSBzaG91bGQgYWx3YXlzIGNhbGN1bGF0ZSB0aGlzIHVzaW5nIGNyb3NzLXZhbGlkYXRpb24uIFlvdSBjYW4gZXZlbiBkbyB0aGUgc3RlcHMgYWJvdmUgMTB4IHdpdGggY3Jvc3MgdmFsaWRhdGlvbiwgc2hvd2luZyByZXNpZHVhbHMgYW5kIHByZWRpY3Rpb25zIG9uIHRoZSB0ZXN0aW5nIGRhdGFzZXQ7IGEgdmVyeSBwcnVkZW50IGFuZCBjYXJlZnVsIG1vZGVsZXIgbWlnaHQgZG8gdGhhdCBpbiBwcmFjdGljZS4KCmBgYHtyfQojRXZhbHVhdGUgdGhlIGFjY3VyYWN5IG9mIHlvdXIgbW9kZWwuIENhbGN1bGF0ZSBhIG1ldHJpYyBmb3IgaXQuCgojIHRoaXMgaXMgdGhlIHN0YXRpc2NhbE1vZGVsaW5nIGZ1bmN0aW9uCiMgdGhleSB3YW50IHlvdSB0byB1c2UgY3Jvc3MgdmFsaWRhdGlvbiBzbyBiYWRseSAoYW5kIHlvdSBzaG91bGQpCiMgdGhhdCB0aGV5IGRvbid0IGhhdmUgYSBzZXBhcmF0ZSBmdW5jdGlvbiBmb3IgTVNFIHdpdGhvdXQgY3Jvc3MtdmFsaWRhdGlvbgojIGN2X3ByZWRfZXJyb3IobW9kZWwpCgojIEhlcmUncyB0aGUgIm5ldyIgd2F5IHdpdGggbW9zYWljTW9kZWwuCgptb2RfZXJyb3IobW9kZWwpICNNU0UKCiMgeW91IGNhbiBsb29rIGF0IGJyb29tKHN1bW1hcnkobW9kZWwpKSB0byBnZXQgCiMgYSBtb3JlIHN0YXRpc3RpY2FsbHkgImFjY3VyYXRlIiBlc3RpbWF0aW9uIG9mIFNFCiMgdGhvdWdoIHRob3NlIG1ldGhvZHMgYXNzdW1lIHRoZSBtb2RlbCdzIGZvcm11bGEgYW5kIGFyY2hpdGVjdHVyZSBtYXRjaCByZWFsaXR5CnNxcnQobW9kX2Vycm9yKG1vZGVsKSkKYGBgCgojIE5leHQsIHdlIHN0YXJ0IGludGVycHJldGluZyB0aGUgbW9kZWwKV2UncmUgZG9pbmcgdGhpcyBhcyBhIHBhcnQgb2Ygc2hvd2luZyB0aGUgbmV3IG1vc2FpY01vZGVsIGZ1bmN0aW9ucywgd2hpY2ggY29ycmVzcG9uZCB0byB0aGUgc3RhdGlzdGljYWxNb2RlbGluZyBmdW5jdGlvbnMgeW91IGxlYXJuZWQgb25saW5lLiBOb3JtYWxseSwgaWYgdGhlIHJlc2lkdWFscyBkaWRuJ3QgbG9vayBnb29kLCB0aGVuIHdlIHdvdWxkbid0IGludGVycHJldCB0aGUgbW9kZWwsIGIvYyB0aGUgZGF0YSBkb2Vzbid0IGFwcGVhciB0byBiZSBmaXQgd2VsbCBieSB0aGUgbW9kZWwgKGl0cyBhc3N1bXB0aW9ucyBhcHBlYXIgdmlvbGF0ZWQpLgpgYGB7cn0KIyBXaGF0IGlzIHRoZSBlZmZlY3Qgc2l6ZSBvZiBlYWNoIGV4cGxhbmF0b3J5IHZhcmlhYmxlIG9uIHRoZSBvdXRjb21lIHZhcmlhYmxlIChkZXBlbmRlbnQgdmFyaWFibGUpPwoKI29sZCB3YXkKIyBlZmZlY3Rfc2l6ZShtb2RlbCwgfiB2YXJpYWJsZXMgdG8gY2hhbmdlKQplZmZlY3Rfc2l6ZShtb2RlbCwgIH4gVFYpCiNuZXcgd2F5Cm1vZF9lZmZlY3QobW9kZWwsIH4gVFYpCm1vZF9lZmZlY3QobW9kZWwsIH4gcmFkaW8pCm1vZF9lZmZlY3QobW9kZWwsIH4gbmV3c3BhcGVyKQoKCiMgaGVyZSdzIHRoZSB1bmRlcmx5aW5nIHdheSBtb2RfZWZmZWN0IHdvcmtzCmVmZmVjdF9zaXplX29mX25ld3NwYXBlciA8LSAoc2FsZXNfbW9kZWxlZChUVj0xNTAsIHJhZGlvPTIzLCBuZXdzcGFwZXIgPSA0NikkbW9kZWxfb3V0cHV0IC1zYWxlc19tb2RlbGVkKFRWPTE1MCwgcmFkaW89MjMsIG5ld3NwYXBlciA9IDI2KSRtb2RlbF9vdXRwdXQpIC8gKDQ2LTI2KQplZmZlY3Rfc2l6ZV9vZl9uZXdzcGFwZXIKCiMgaGVyZSdzIGhvdyB0byBnZXQgdGhlIGZ1bmN0aW9uIGZvciB0aGUgbW9kZWwKIyBpLmUuIGZ1bmN0aW9uKGV4cGxhbmF0b3J5IHZhcmlhYmxlcykgd2hpY2ggZ2l2ZXMgYSBwcmVkaWN0aW9uIGZvciBkZXBlbmRlbnQgdmFyaWFibGUKc2FsZXNfbW9kZWxlZCA8LSBtb2RfZnVuKG1vZGVsKQoKCmBgYAoKCiMjIFNrZXB0aWNhbCBNb2RlbGluZyBwcm9jZXNzCkhlcmUncyBhIHByb2Nlc3MgdG8gZm9sbG93IHdoZW4gbW9kZWxpbmcgKGFzIGEgcmVmcmVzaGVyKS4gV2Ugd29uJ3QgY292ZXIgYWxsIHRoZSBzdGVwcyBoZXJlIGluIHRoaXMgYW5hbHlzaXMuCgoxLiBFeHBsb3JhdG9yeSBkYXRhIGFuYWx5c2lzCgpTcGxpdCBkYXRhIGludG8gdHJhaW5pbmcgYW5kIHRlc3QKCjIuIEZpdCBhIG1vZGVsCjMuIEV2YWx1YXRlIHRoZSBtb2RlbCBieSBhZGRpbmcgcmVzaWR1YWxzIGFuZCBsb29raW5nIGF0IE1TRQogIFVzaW5nIGNyb3NzLXZhbGlkYXRpb24KICAKSW50ZXJwcmV0IG1vZGVsCiAgVXNpbmcgYm9vdHN0cmFwcGVkIGNvbmZpZGVuY2UgaW50ZXJ2YWxzIGZvciBlZmZlY3Qgc2l6ZXMgLyBwYXJhbWV0ZXJzCiAgCjQuIFVzZSBtb2RlbCBmb3IgZGVjaXNpb24KCiMgQ29udGludWluZyBvdXIgYW5hbHlzaXMKQXMgdGhlIG1vZGVsIGFib3ZlIGRpZG4ndCBzZWVtIHRvIGZpdCB0aGUgZGF0YSwgbGV0J3MgdHJ5IG9uZSB3aXRoIGFuIGludGVyYWN0aW9uIHRlcm0sIG1vdGl2YXRlZCBieSBvdXIgZG9tYWluIGtub3dsZWRnZS4KCmBgYHtyfQoKIyBVc2luZyBtdWx0aXBsZSByZWdyZXNzaW9uIHdpdGggaW50ZXJhY3Rpb24gdGVybXM6Cgptb2RlbF8yIDwtIGxtICggc2FsZXMgfiBUViAqIHJhZGlvICsgbmV3c3BhcGVyLCBkYXRhPWFkc19zYWxlcykKbW9kZWxfMgoKIyBsZXQncyBjb21wYXJlIHRoZSB0d28gbW9kZWxzLCB0aGUgbmV3IG9uZSBoYXMgaW50ZXJhY3Rpb24gZWZmZWN0cwpncmlkLmFycmFuZ2UobW9kX3Bsb3QobW9kZWwpK2xhYnMoc3VidGl0bGU9Ik5vIGludGVyYWN0aW9ucyBtb2RlbCIpICwKICAgICAgICAgICAgIG1vZF9wbG90KG1vZGVsXzIpK2xhYnMoc3VidGl0bGU9IlRWKnJhZGlvICsgbmV3c3BhcGVyIG1vZGVsICh3aXRoIGludGVyYWN0aW9uKSIpLCBuY29sPTEpCgoKCgojIFdoYXQgaXMgdGhlIGVmZmVjdCBzaXplIG9mIGVhY2ggZXhwbGFuYXRvcnkgdmFyaWFibGUgb24gdGhlIG91dGNvbWUgdmFyaWFibGUgKGRlcGVuZGVudCB2YXJpYWJsZSk/Cgptb2RfZWZmZWN0KG1vZGVsLCB+IFRWKSAjIGxpa2UgZWZmZWN0X3NpemUKbW9kX2VmZmVjdChtb2RlbF8yLCB+IFRWKSAjIGxpa2UgZWZmZWN0X3NpemUKbW9kX2VmZmVjdChtb2RlbF8yLCB+IHJhZGlvKQptb2RfZWZmZWN0KG1vZGVsXzIsIH4gbmV3c3BhcGVyKQoKc2FsZXNfbW9kZWxfMmVkIDwtIG1vZF9mdW4obW9kZWxfMikgCiMgc2FsZXMgPSBhMSArIGEyKnR2ICsgYTMqcmFkaW8gKyBhNCp0dipyYWRpbwoKCmVmZmVjdF9zaXplX29mX25ld3NwYXBlciA8LSAoc2FsZXNfbW9kZWxfMmVkKFRWPTE1MCwgcmFkaW89MjMsIG5ld3NwYXBlciA9IDQ2KSRtb2RlbF8yX291dHB1dCAtc2FsZXNfbW9kZWxfMmVkKFRWPTE1MCwgcmFkaW89MjMsIG5ld3NwYXBlciA9IDI2KSRtb2RlbF8yX291dHB1dCkgLyAoNDYtMjYpCmVmZmVjdF9zaXplX29mX25ld3NwYXBlcgojIEhpbnQ6IHlvdSBjYW4gdXNlIHRvb2xzIGZyb20gc3RhdGlzdGljYWxNb2RlbGluZyBwYWNrYWdlIHVzZWQgaW4gdGhlIERhdGFjYW1wIGNvdXJzZQoKIyBFdmFsdWF0ZSB0aGUgYWNjdXJhY3kgb2YgeW91ciBtb2RlbF8yLiBDYWxjdWxhdGUgYSBtZXRyaWMgZm9yIGl0LgpzcXJ0KG1vZF9lcnJvcihtb2RlbF8yKSkKaGlzdChhZHNfc2FsZXMkc2FsZXMpCiMgSW50ZXJwcmV0IHRoZSBtZXRyaWMgZm9yIGFjY3VyYWN5IGFib3ZlLgpzdW1tYXJ5KG1vZGVsXzIpCgpzYWxlc193aXRoX3Jlc2lkdWFsc18yIDwtIGF1Z21lbnQobW9kZWxfMiwgYWRzX3NhbGVzKQpzYWxlc193aXRoX3Jlc2lkdWFsc18yCmcgPC0gZ2dwbG90KGRhdGE9c2FsZXNfd2l0aF9yZXNpZHVhbHNfMiwgYWVzKHNhbGVzKSkgCmcgKyBnZW9tX2hpc3RvZ3JhbSggCiAgICAgICAgICAgICAgICAgICBiaW53aWR0aCA9IC41LCAKICAgICAgICAgICAgICAgICAgIGNvbD0iYmxhY2siLCAKICAgICAgICAgICAgICAgICAgIHNpemU9LjEpICsgeGxpbShjKDAsIDMwKSkKCmcgPC0gZ2dwbG90KGRhdGE9c2FsZXNfd2l0aF9yZXNpZHVhbHNfMiwgYWVzKC5maXR0ZWQpKSAKZyArIGdlb21faGlzdG9ncmFtKCAKICAgICAgICAgICAgICAgICAgIGJpbndpZHRoID0gLjUsIAogICAgICAgICAgICAgICAgICAgY29sPSJibGFjayIsIAogICAgICAgICAgICAgICAgICAgc2l6ZT0uMSkgKyB4bGltKGMoMCwgMzApKQoKZyA8LSBnZ3Bsb3QoZGF0YT1zYWxlc193aXRoX3Jlc2lkdWFscywgYWVzKC5maXR0ZWQpKSAKZyArIGdlb21faGlzdG9ncmFtKCAKICAgICAgICAgICAgICAgICAgIGJpbndpZHRoID0gLjUsIAogICAgICAgICAgICAgICAgICAgY29sPSJibGFjayIsIAogICAgICAgICAgICAgICAgICAgc2l6ZT0uMSkgKyB4bGltKGMoMCwgMzApKSArIGxhYnMoc3VidGl0bGU9Ik5vIGludGVyYWN0aW9uIG1vZGVsIikKICAKc2FsZXNfd2l0aF9yZXNpZHVhbHNfMiA8LSBhdWdtZW50KG1vZGVsXzIsIGFkc19zYWxlcykKc2FsZXNfd2l0aF9yZXNpZHVhbHNfMgoKCiMgcmVzaWR1YWxzIGhpc3RvZ3JhbXMKcGxvdF9saW1pdCA8LSBtYXgoYWJzKHNhbGVzX3dpdGhfcmVzaWR1YWxzXzIkLnJlc2lkKSkrLjUKZyA8LSBnZ3Bsb3QoZGF0YT1zYWxlc193aXRoX3Jlc2lkdWFsc18yLCBhZXMoLnJlc2lkKSkgCnNhbGVzX3Jlc2lkdWFsc19wbG90XzIgPC0gZyArIGdlb21faGlzdG9ncmFtKCAKICAgICAgICAgICAgICAgICAgIGJpbndpZHRoID0gLjUsIAogICAgICAgICAgICAgICAgICAgY29sPSJibGFjayIsIAogICAgICAgICAgICAgICAgICAgc2l6ZT0uMSkgKyB4bGltKGMoLXBsb3RfbGltaXQsIHBsb3RfbGltaXQpKSArIGxhYnMoc3VidGl0bGU9IiBIYXMgaW50ZXJhY3Rpb24gbW9kZWwiKQpzYWxlc19yZXNpZHVhbHNfcGxvdF8yCgojIGNvbXBhcmUgdG8gc2ltdWxhdGVkIHJlc2lkdWFscwpzZDIgPC0gc2lnbWEobW9kZWxfMikKc2QyCiMgdGhlIGRvKG4pIHN5bnRheCBpcyBhIHNob3J0Y3V0IGZvcgojIGRvaW5nIHRoZSBmb2xsb3dpbmcgcGFydHMgbiB0aW1lcywgdGhlbiBjb21iaW5pbmcgdGhlIHJlc3VsdHMgaW50byBhIHZlY3Rvcgpucm93KGFkc19zYWxlcykKc2ltdWxhdGVkX3Jlc2lkdWFsc19mb3JfZ2F1c3NpYW5fbm9pc2UyIDwtIGRvKG5yb3coYWRzX3NhbGVzKSkgKiBybm9ybSgxLDAsc2QyKQpzaW11bGF0ZWRfcmVzaWR1YWxzX2Zvcl9nYXVzc2lhbl9ub2lzZTIKCmcgPC0gZ2dwbG90KGRhdGE9c2ltdWxhdGVkX3Jlc2lkdWFsc19mb3JfZ2F1c3NpYW5fbm9pc2UyLCBhZXMocm5vcm0pKSAKZyA8LSBnICsgZ2VvbV9oaXN0b2dyYW0oIAogICAgICAgICAgICAgICAgICAgYmlud2lkdGggPSAuNSwgCiAgICAgICAgICAgICAgICAgICBjb2w9ImJsYWNrIiwgCiAgICAgICAgICAgICAgICAgICBzaXplPS4xKSArIHhsaW0oYygtcGxvdF9saW1pdCwgcGxvdF9saW1pdCkpICsgbGFicyhzdWJ0aXRsZT0iU2ltdWxhdGVkIEdhdXNzaWFuIG5vaXNlIikKCmdyaWQuYXJyYW5nZShnLHNhbGVzX3Jlc2lkdWFsc19wbG90XzIsbmNvbD0xKQoKI2NvbXBhcmUgdG8gdGhlIG5vIGludGVyYWN0aW9uIG1vZGVsCmdyaWQuYXJyYW5nZShzaW11bGF0ZWRfcmVzaWR1YWxzX2Zvcl9nYXVzc2lhbl9ub2lzZV8xX3Bsb3Qsc2FsZXNfcmVzaWR1YWxzX3Bsb3QsbmNvbD0xKQoKIyB5b3UgY2FuIHNlZSB0aGF0IG91ciBuZXcgbW9kZWwgaGFzIGJldHRlciBsb29raW5nIHJlc2lkdWFscywgbW9yZSBzaW1pbGFyIHRvIAojIGdhdXNzaWFuIG5vaXNlCgojIG1vZGVsaW5nIHByb2NlZWRzIGluIGEgc2VxdWVuY2UgbGlrZSB0aGlzCiMgMS4gdHJ5IG1vZGVsLCBldmFsdWF0ZSBpdHMgZml0IAojIDIuIGNoYW5nZSBtb2RlbCBhcmNoaXRlY3R1cmUgYW5kIGZvcm11bGEgd2hlbiB5b3Ugc2VlIHBhdHRlcm5zIGluIHJlc2lkdWFscwojIDMuIHJlcGVhdCwgdG8gdGhlIHF1YWxpdHkgb2YgbW9kZWwgeW91IG5lZWQgZm9yIHlvdXIgcHVycG9zZSAoc3RyaXZlIGZvciBoaWdoZXIgdGhvKQpgYGAKIyNBc3Nlc3MgcHJlZGljdGlvbnMKSGVyZSBhcmUgc29tZSBidWlsdC1pbiBwbG90cyB0aGF0IGFsbCBoYXZlIHRoZSBzYW1lIHB1cnBvc2UgLSB0byB2YWxpZGF0ZSB0aGF0IHRoZSBtb2RlbCBmaXRzIHlvdXIgZGF0YS4gSSB3b24ndCBnbyBpbiBkZXB0aCB3aXRoIHRoZXNlLCBidXQgdGhlIGdlbmVyYWwgaWRlYSBvZiBlYWNoIGlzIHRoYXQgeW91IHdhbnQgdGhlIHBvaW50cyB0byBmYWxsIGFyb3VuZCB0aGUgZG90dGVkIGxpbmUgd2l0aCBqdXN0IHJhbmRvbSB2YXJpYXRpb24gYXJvdW5kIGl0IHdpdGhvdXQgYW55IHBhdHRlcm5zIChvbiBxLXEgcGxvdCAsIHRoZSAybmQgb25lIGZyb20gdGhlIHBsb3QgY2FsbCwgeW91IHdhbnQgdGhlbSB0byBmYWxsIG9uIHRoZSBsaW5lKS4KClRoZXJlIGFyZSBhbHNvIHN0YXRpc3RpY2FsIHRlc3RzIGZvciBub3JtYWxpdHkgb2YgcmVzaWR1YWxzLCBidXQgaW4gcHJhY3RpY2UgYWxtb3N0IG5vIG9uZSB1c2VzIHRoZW0gYmVjYXVzZSAxKSB0aGV5IGFsbW9zdCBhbHdheXMgc2F5IHRoYXQgdGhlIHJlc2lkdWFscyBhcmUgbm90IG5vcm1hbCAod2hpY2ggaXMgYSBibG93IHRvIHlvdXIgbW9kZWwgc28gdGhleSBqdXN0IGlnbm9yZSB0aGF0Li4uKSAyKSBJIGhhdmUgaGVhcmQgcGVvcGxlIHNheSB0aGF0IHRoZSBsaW5lYXIgbW9kZWwgaGFzIHNvbWUgInJvYnVzdG5lc3MiIGFnYWluc3QgZGV2aWF0aW9ucyBmcm9tIGl0cyBhc3N1bXB0aW9ucywgYnV0IEkgZG9uJ3QgcmVhbGx5IGJ1eSB0aGF0IGIvYyBpZiB3ZSB1bmRlcnN0b29kIHRoZSByb2J1c3RuZXNzIGNvbmRpdGlvbnMgd2VsbCwgd2Ugc2hvdWxkIGJlIGFibGUgdG8gbWFrZSBhIHRlc3QgZm9yICJub3JtYWwgZW5vdWdoIiByZXNpZHVhbHMsIGFuZCBJJ3ZlIG5ldmVyIHNlZW4gb25lIG9yIGhlYXJkIG9uZSBtZW50aW9uZWQuIFRodXMsIEknbSBlbXBoYXNpemluZyB0aGUgdmlzdWFsIGFwcHJvYWNoICh3aGljaCBpcyBhbHNvIG1vcmUgaW5mb3JtYXRpdmUgdGhhbiBhIHAtdmFsdWUpLgoKYGBge3J9CnBsb3QobW9kZWxfMikKCiNjb21wYXJlIHRvIG1vZGVsIHdpdGhvdXQgaW50ZXJhY3Rpb25zCnBsb3QobW9kZWwpCgpgYGAKIyBDb21tZW50YXJ5IG9uICJvdXRsaWVycyIgYW5kIENvb2sncyBkaXN0YW5jZQpJJ20gZ29pbmcgdG8gdXNlIHRoaXMgbGFzdCBwbG90IG9mIHRoZSA0IHRvIG1ha2UgYSB2ZXJ5IGltcG9ydGFudCBwb2ludCAtIGRvbid0IHJlbW92ZSBvdXRsaWVycyBmcm9tIHlvdXIgZGF0YSB0aGUgYWN0dWFsbHkgcmVmbGVjdCByZWFsaXR5LiBUaGUgbGFzdCBwbG90IHdpdGggbGV2ZXJhZ2Ugc2hvd3MgcG9pbnRzIHRoYXQgaGF2ZSBhbiBvdmVybHkgbGFyZ2UgaW5mbHVlbmNlIG9uIHRoZSBmaXR0ZWQgbGluZSAocmVjYWxsIHRoYXQgbGluZWFyIG1vZGVscyBtaW5pbWl6ZSBzdW0gb2YgdGhlIHNxdWFyZXMgb2YgdGhlIHJlc2lkdWFscywgYW5kIGFuIG91dGxpZXIgcG9pbnQgaXMgZ29pbmcgdG8gYmUgZXZlbiBiaWdnZXIgd2hlbiBzcXVhcmVkLCBzbyBpdCBoYXMgbW9yZSBpbmZsdWVuY2UpLiBUaGlzIGlzIG9uZSBvZiB0aGUgbWFpbiBpc3N1ZXMgd2l0aCBsaW5lYXIgbW9kZWxzLCBzbyBwZW9wbGUgb2Z0ZW4gdHJ5IHRvIHRyaW0gdGhlaXIgZGF0YSBvZiBvdXRsaWVycyBhbmQgbWF5IHVzZSBjb29rJ3MgZGlzdGFuY2UgdG8gYXJndWUgdGhhdC4KCkhvd2V2ZXIsIGl0J3MgaW1wcm9wZXIgdG8gcmVtb3ZlIGRhdGEgdW5sZXNzIHlvdSByZWFsbHkgdW5kZXJzdGFuZCBpdCBhbmQgc2VlIGl0IHdhcyBhIG1lYXN1cmVtZW50IGVycm9yOyB5b3Ugc2hvdWxkbid0IHJlbW92ZSByZWFsaXR5IGZyb20geW91ciBkYXRhc2V0LiBPdGhlcndpc2UgeW91IGNhbiBtYWtlIHZlcnkgYmlnIG1pc3Rha2VzIChleGFtcGxlIG9mIGZhdWx0eSBsb2dpYzogImxldCdzIGVzdGltYXRlIHRoZSBlZmZlY3RzIG9mIHdhciBvbiBjb3VudHJpZXMsIG9oLCB3b3JsZCB3YXIgMSBhbmQgd29ybGQgd2FyIDIgd2VyZSBqdXN0IG91dGxpZXJzLCB3ZSBjYW4gZXhjbHVkZSB0aG9zZS4iIFRoYXQgbG9naWMgd2lsbCBsZWFkIHRvIHNvbWUgbGFyZ2UgdW5kZXJlc3RpbWF0ZXMgb2YgdGhlIHBvdGVudGlhbCBlZmZlY3RzIG9mIHdhcikuIEV2ZW4gdGhvdWdoIHRoZSB3YXIgZXhhbXBsZSBzaG93cyB0aGUgZm9vbGlzaG5lc3Mgb2YgdGhpcyBsb2dpYywgc2NpZW50aXN0cyBhbmQgZXZlbiBzdGF0aXN0aWNpYW5zIGluIHByYWN0aWNlIHNvbWV0aW1lcyB1c2UgdGhhdCBmbGF3ZWQgbG9naWMuCgpOb3csIHNvbWV0aW1lcyBwZW9wbGUgYWxzbyB0cnkgdG8gYXJndWUgcmVtb3Zpbmcgb3V0bGllcnMgdXNpbmcgbGFuZ3VhZ2UgZ2FtZXMuIFN0YXRpc3RpY2lhbnMgYW5kIHNjaWVudGlzdHMgbWlnaHQgdHJ5IHRvIGp1c3RpZnkgcmVtb3Zpbmcgb3V0bGllcnMgYnkgc2F5aW5nIHRoZXkgYXJlIHN0dWR5aW5nICJzbWFsbCBjb25mbGljdHMiIG5vdCB3YXJzIC0ganVzdCBjaGFuZ2luZyB0aGUgd29yZHMgdGhleSB1c2UgdG8gcmVmZXIgdG8gdGhlaXIgcGhlbm9tZW5hLiBUaGUgZmFjdCBpcyB0aGF0IHRoZSBkeW5hbWljcyB0aGF0IHVuZGVybHkgY29uZmxpY3QgcHJvZHVjZSBtYW55IHNtYWxsIGV2ZW50cyBhcyB3ZWxsIGFzIHJhcmUgZXh0cmVtZSBldmVudHM7IGV4Y2x1ZGluZyB0aG9zZSByYXJlIGV2ZW50cyBpbiB5b3VyIG1vZGVscyBsZWFkcyB0byB0aGluZ3MgbGlrZSAxKSBmaW5hbmNpYWwgY3Jpc2VzLCBhbmQgMikgYnVpbGRpbmcgd2FsbHMgYXJvdW5kIEZ1a3VzaGltYSBudWNsZWFyIHJlYWN0b3JzIHRoYXQgYXJlIG5vdCBoaWdoIGVub3VnaC4gCgogIA==